Implementing Metcast in Scheme

This paper presents a case study of implementing a large distributed system in Scheme. Metcast is a request-reply and subscription system for dissemination of real-time weather information. The system stores a large amount of weather observation reports, forecasts, gridded data produced by weather models, and satellite imagery. A Metcast server delivers a subset of these data in response to a query formulated in a domain-speci c language. Decoders of World Meteorological Organization's data feed, the Metcast server, XML encoders and decoders, auxiliary and monitoring CGI scripts are all written in Scheme. This paper considers two examples that demonstrate bene ts of our choice of the implementation language: parsing of the data feed and a module system for the Metcast server. We will also discuss extensions to Scheme as well as performance. 1 Overview of Metcast Metcast is a request-reply and a subscription system for distributing, disseminating, publishing and broadcasting of real-time weather information [1]. The system comprises clients and servers communicating in an HTTP protocol. A Metcast server maintains a database of weather observation reports, forecasts, advisories, gridded data produced by weather models, as well as of satellite imagery and plain text messages and discussions. A Metcast client uses a web form or a domain-speci c, exible request language to retrieve a subset of data from a Metcast database [2]. A Metcast server which is an application (web) server parses requests, queries the database and sends the requested data in a singleor a multi-part reply. A server may act as a client to request a subset of data for further redistribution. Metcast servers are in operation on several U.S. Navy Meteorology and Oceanography centers worldwide. Clients are deployed on great many sites throughout the U.S. Navy as well as U.S. Air Force, DoD, NATO, NOAA and other government agencies. One particular source of original data is World Meteorological Organization's (WMO) data feed, containing a great number of land and sea surface and depth/height prole reports, forecasts, advisories, discussions, etc. for the whole globe. A set of decoders processes the feed, and stores This work has been supported by SPAWAR PMW-185, FNMOC and in part by the National Research Council, Naval Postgraduate School, and the Army Research O ce under contracts 38690-MA and 40473-MA-SP. raw and decoded data in a database. A Metcast server distributes this information in an XML OMF format [3]. The Metcast server, the set of decoders for various WMO data formats, auxiliary and monitoring CGI scripts are all written in Scheme. Metcast clients are written in C++, Java, Scheme, Perl, Python, JavaScript, and Visual Basic. The server and related modules are implemented in 12800 lines of Scheme code, counting the comments. WMO data feed decoders add 8400 more lines. The size of common extension libraries is 5400 lines of Scheme and some embedded C code. A Gambit-C 3.0 Scheme interpreter enhanced with compiled-in extensions has been used throughout the project. 2 Parsing of the data feed Scheme proved to be particularly helpful in parsing of the WMO data feed. WMO code is a rather old, ad hoc, peculiar, somewhat inconsistent, tangled data format with a number of options, exceptions and special cases. Furthermore, received bulletins often contain errors due to manual miscoding and transmission problems. A typical WMO report for example, a surface synoptic report is a sequence of code groups separated by white space. A code group is a string of letters, numbers and a few special characters. A code group or groups encode the result of observation of a particular quantity, e.g., cloud conditions, temperature, etc. If code groups were atomic tokens, a report could easily be parsed by a LR(1) automaton. Alas, code groups are composite entities that encode information in idiosyncratic ways. The mere identi cation of a code group depends on its position and context, which may encompass all previously seen code groups. We have implemented a report decoder as a combination of a table-driven automaton and code-based group parsers. The latter recognize, parse, and validate a particular code group. The decoder takes a list of code groups and returns an associative list, an Abstract Syntax "Tree" (AST). A special procedure later walks the AST and records the parsed data in a database upload bu er. Of a particular help was Scheme's ability to store and pass procedural values as any other values. This let us implement decoders as compositions of code group parsers. For example, a very typical production ? * ? can be parsed by a combination (sequence parse-a (sequence (loop parse-b) parse-c)). This composition of group parsers is represented by a list (parse-a (repetition-flag parse-b) parse-c). Given this list and the list of code groups to decode, a main driver walks both lists, applying the current parser to the current code group. The result of the application as well as the repetition ag determine if the current code group is consumed, if the next parser should be chosen, and how AST should be extended. All the group parsers have the same interface. They receive as arguments the current code group and the AST, and should return: an association (a name-value pair) or a list of such associations to add to the AST; a symbol pass if the parser failed to recognize the code group. The code group should be given to the next parser; #f meaning a syntax error is detected at the current token; a symbol terminate to stop parsing of the report. In the successful case (the rst one above), the current token is assumed consumed. Any group parser may examine the AST (that is, the results of the previous parsers) and may even modify the AST. Therefore our parsing technique is somewhat similar to attribute grammars. Figure 1 shows an example of a group parser. The example demonstrates an and-let* construction (srfi2), which was used frequently throughout the project and proved very helpful. As Fig. 1 shows, once the current token has been recognized as a potential group, and-let* carries on a sequence of elementary parsing decisions, all of which must succeed. The Metcast decoder is continually processing incoming les, which are delivered every 1-3 minutes. A rather large batch of reports 8 plain-text bulletins, 144 sea surface observation reports, 777 upper-air level data, 2 terminal airdrome forecasts and 322 synoptic reports takes 8 wall-clock seconds to parse and 19 seconds to upload and record into the database. The platform is Sun Enterprise-450 server with two Ultrasparc-ii cpus and 512 mb ram, running Solaris 2.6 and Informix 7.3 database. Keeping in mind that incoming reports have up to 10-minute delay from the time of issue, the total processing time at the Metcast end under 1 minute is entirely acceptable. 3 Implementing the Web application server Scheme turned out to be a good implementation language for a web application server as well. One part of the server is a complex nite state machine that decides when a multipart reply is called for, and sends the corresponding mime headers. The problem is not trivial as it is generally impossible to predict the number of non-empty replies for a complex request. Expressing such nite automata as sets of mutually-recursive procedures made the code clear and exible. Scheme was conducive to compilation and interpretation of the S-expression-based Metcast Request Language [2]. A request language phrase is compiled into a dictionary an ordered sequence of bindings, which constitutes the environment to look up all data needed to construct a Metcast database query. This hierarchical repository follows neither the static scope of Scheme expressions, nor the dynamic scope of procedure activations. Some bindings may be to procedures, which may push additional associations into the environment and thus a ect further lookups. Metcast server has a highly modular structure. The main program is responsible for receiving and parsing of a request, and packing of replies. Execution of a particular product request is delegated to a separate module (plug-in). The hierarchical repository was indispensable in implementing a parameter bus, which maintains the con guration for the main server and all plug-ins. The parameter bus also provides a uniform interface for invocation of modules and passing of a complex set of explicit and default parameters. For example, the main Metcast server module contains a form (include "metar.scm") that loads a plug-in metar.scm. The latter le de nes procedures perform-metar-request and performMSL-request. The le binds these procedures to the corresponding Request Language verbs and the con guration information: